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Executive  Summary 


This  paper  is  motivated  by  a  2014  Institute  for  Defense  Analyses  (DA)  paper1 
(hereafter  McN-W  (2014))  on  whether  there  is  a  statistically  significant  association 
between  cost  growth  and  changes  over  time  in  acquisition  policies  and  processes,  after 
taking  funding  climate  into  account.  McN-W  (2014)  found  that  there  is  not  a  statistically 
significant  association  between  cost  growth  and  the  changes  in  acquisition  policy  and 
process.  There  was,  however,  a  statistically  significant  association  between  cost  growth 
and  funding  climate.  In  particular,  average  cost  growth  was  much  higher  for  Major 
Defense  Acquisition  Programs  (MDAPs)  that  entered  what  is  now  called  System  Design 
and  Development  (SDD)  in  “bust”  phases  of  the  “boom-bust”  cycles  in  Department  of 
Defense  (DoD)  funding. 

McN-W  (2014)  used  cost  growth  data  only  for  MDAPs  that  completed  development 
and  went  into  production.  This  paper  extends  the  analysis  to  MDAPs  that  were  cancelled. 
The  program  outcome  metric  used,  however,  is  cancellations,  not  cost  growth. 

The  paper  begins  with  a  brief  survey  of  MDAP  cancellations,  which  is  not  available 
in  the  literature.  The  next  section  asks  whether  very  high  cost  growth  is  the  main  cause  of 
most  cancellations.  These  two  “scene  setters”  open  the  door  to  a  deeper  look  at  the  data. 
They  are  followed  by  a  section  that  states  the  plan  of  the  analysis  and  then  the  two  main 
analytical  sections  and  conclusions. 

A  Brief  Census  of  Cancellations 

The  population  used  in  this  paper  was  the  set  of  278  MDAPs  that  passed  Milestone 
(MS)  B  or,  prior  to  2000,  MS  II  during  1970-2007.  Of  these,  fifty-three — -just  under  20 
percent  of  the  total — were  cancelled.  Cancellation  rates  were  roughly  comparable  across 
the  Services,  although  joint  programs  were  somewhat  more  likely  to  be  cancelled.  Forty- 
three  percent  of  the  cancellations  were  initiated  by  the  Office  of  the  Secretary  of  Defense 
and  34  percent  by  the  Services.  The  remaining  cancellations  were  initiated  by  the  White 
House  or  the  Congress.  The  median  time  from  MS  II/B  to  cancellation  was  4.3  years  and 
the  average  time,  5.2  years.  Only  eight  of  the  programs  cancelled  were  in  Low  Rate 
Initial  Production,  and  only  one  was  in  Full  Rate  Production. 
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Cost  Growth  and  Cancellations 

It  seems  to  be  the  conventional  wisdom  that  cost  growth  is  a  major  cause  of  most 
program  cancellations.  There  are  some  well-known  instances  in  which  this  was  the  case. 
The  literature,  however,  does  not  establish  that  it  is  a  typical  cause  of  program 
cancellations.  That  is  a  point  that  needs  to  be  checked  and  at  least  to  some  extent  can  be. 

We  were  able  to  estimate  cost  growth  for  twenty-five  of  the  fifty-three  MDAPs  that 
were  cancelled.  The  database  used  for  McN-W  (2014)  provides  cost  growth  estimates  for 
an  additional  151  MDAPs  that  were  not  cancelled,  for  a  total  of  176  programs.  Fifty  of 
these  176  programs  showed  growth  of  at  least  50  percent.  Of  these,  ten  were  cancelled 
and  forty  were  not.  The  sample  cancellation  rate  for  programs  with  cost  growth  of  at  least 
50  percent  is  then  20  percent.  The  cancellation  rate  for  programs  with  cost  growth  of  less 
than  50  percent  is  much  less — about  12  percent.  The  key  point  for  present  purposes, 
however,  is  that  80  percent  of  the  MDAPs  with  cost  growth  estimates  of  at  least  50 
percent  were  not  cancelled.  In  short,  there  is  more  than  cost  growth  influencing  MDAP 
cancellations. 

Plan  of  the  Analysis 

The  question  asked  is  whether  the  different  acquisition  regimes  and  funding 
climates  have  a  statistically  discernible  association  with  cancellation  rates.  Each  of  two 
cancellation  rates  was  considered: 

•  The  cancellation  rate  for  the  cohort  of  MDAPs  that  passed  MS  II/B  during  a 
period  of  years.  For  example,  45  MDAPs  passed  MS  II  during  Fiscal  Year 
(FY)  1994-FY  2000,  of  which  6  were  eventually  cancelled,  for  a  cohort 
cancellation  rate  of  13  percent. 

•  The  average  number  of  cancellations  over  a  span  of  years.  For  example,  in  the 
four  years  FY  1986-FY  1989,  nine  MDAPs  were  cancelled,  for  an  average 
cancellation  rate  of  2.3/yr. 

This  paper  distinguishes  five  successive  DoD  acquisition  regimes: 

1.  Defense  Systems  Acquisition  Review  Council  (DSARC),  1970-1982 

2.  Post-Carlucci  Initiatives  DSARC,  1983-1989 

3.  Defense  Acquisition  Board  (DAB),  1990-1993 

4.  Acquisition  Reform  (AR),  1994-2000 

5.  DAB  post  AR,  2001-2012 
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There  were  two  complete  bust-boom  cycles  during  the  period  1970-2008: 

•  The  reduction  in  DoD  funding  associated  with  the  winding  down  of  US 
involvement  in  the  War  in  Vietnam  through  the  end  of  the  Carter-Reagan 
buildup,  roughly  1970-1986. 

•  The  post-Cold  War  reduction  in  DoD  funding  and  the  subsequent  buildup 
associated  with  the  wars  in  Afghanistan  and  Iraq,  roughly  1987-2008. 

The  start  of  the  withdrawal  of  US  forces  from  Iraq  in  2009  marked  the  start  of  a  third 
cycle,  which  was  ratified  by  passage  of  the  Budget  Control  Act  (BCA)  in  20 1 1 . 

Changes  over  time  in  acquisition  regime  and  the  boom-bust  cycle  in  DoD  funding 
provide  natural  experiments  that  may  shed  some  light  on  the  effects  of  acquisition  policy 
and  funding  climate  on  cancellation  rates.  For  example,  looking  at  changes  in  the  cohort 
cancellation  rate  across  successive  acquisition  regimes  during  bust  phases  may  reveal  the 
effects  of  changes  in  acquisition  regime.  While  such  analyses  can  be  useful,  it  is 
necessary  to  recognize  that,  in  addition  to  the  easily  identified  elements  of  the  natural 
experiment,  there  are  other  factors  that  may  have  influenced  cohort  cancellation  rates.  We 
first  state  the  statistical  results  for  the  basic  natural  experiments  and  then  ask  whether 
they  are  compromised  by  the  omission  of  other  factors. 

Analysis  of  Cohort  Cancellation  Rates 

Both  bust  and  boom  conditions  were  experienced  during  three  of  the  acquisition 
regimes.  For  two  of  these,  the  statistical  analysis  indicated  that  the  average  cancellation 
rates  of  cohorts  that  passed  MS  II/B  in  bust  periods  were  not  significantly  different  from 
those  of  cohorts  that  passed  MS  II/B  in  boom  periods.  The  difference  was  significant, 
however,  for  the  Post-Carlucci  DSARC,  which  had  a  very  high  average  cancellation  rate 
for  the  cohort  that  passed  MS  II  in  the  bust  years  FY  1987-FY  1989  (42  percent)  and  a 
comparatively  low  cancellation  rate  for  the  cohort  of  FY  1983-FY  1986  (12  percent). 
The  average  cohort  cancellation  rate  under  the  Post-Carlucci  DSARC  also  was 
significantly  higher  than  those  of  the  other  acquisition  regimes  in  the  bust  phase  of  the 
cycle.  Two  factors  not  considered  in  the  statistical  analysis  may  have  influenced  the 
results: 

•  The  end  of  the  Cold  War;  and 

•  An  apparent  reduced  emphasis  during  the  second  Reagan  Administration  on  the 
policy  of  not  starting  more  MDAPs  than  likely  future  budgets  could  sustain. 

The  conclusion  reached  depends  on  the  weight  given  to  these  historical  considerations.  If 
they  are  discounted,  the  conclusion  would  be  that  during  the  Post-Carlucci  DSARC 
regime,  the  effect  of  funding  climate  on  the  average  cohort  cancellation  rate  was 
statistically  significant  and  significantly  higher  for  this  regime  than  for  other  acquisition 
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regimes.  If,  instead,  the  historical  factors  are  heavily  weighted,  the  conclusion  would  be 
that  neither  acquisition  regime  nor  funding  climate  mattered  for  cohort  cancellation  rates. 
The  difference  between  these  two  positions  is  thin.  A  reasonable  summary  conclusion  is 
that  neither  acquisition  policy  and  process  changes  nor  changes  in  funding  climate  have 
had  much,  if  any,  effect  on  average  cohort  cancellation  rates. 

Analysis  of  Annual  Cancellation  Rates 

We  found  that  the  budget  climates  relevant  to  analysis  of  annual  cancellation  rates 
were  (1)  years  in  which  the  funds  appropriated  for  DoD  were  increasing  or  stable;  and 
(2)  periods  in  which  DoD  funding  decreased  significantly  from  one  year  to  the  next.  The 
average  annual  cancellation  rates  for  this  partitioning  of  funding  climates  appear  in  the 
table  below.  Note  that  the  average  annual  cancellation  rates  are  comparatively  high  in 
periods  when  DoD  funding  was  sharply  declining  and  comparatively  low  in  periods  in 
which  funding  was  stable  or  increasing. 


Average  Number  of  Cancellations  per  Fiscal  Year  by  Acquisition  Regime 

and  Funding  Climate 


Sharply  Decreasing  Funding 

Stable  or  Increasing  Funding 

Acquisition  Regime 

Period 

Cancellation  Rate 

Period 

Cancellation  Rate 

DSARC 

1970-1982 

0.5/yr. 

(6  in  13  yrs.) 

Post-Carlucci  DSARC 

1986-1989 

2.3/yr. 

(9  in  4  yrs.) 

1983-1985 

0.3/yr. 

(1  in  3  yrs.) 

DAB 

1990-1993 

3.3/yr. 

(13  in  4  yrs.) 

AR 

1994-2000 

0.4/yr. 

(3  in  7  yrs.) 

DAB  post  AR 

2009-2012 

2.8/yr. 

(11  in  4  yrs.) 

2001-2008 

1 ,3/yr. 

(10  in  8  yrs.) 

Note:  Number  of  cancellations  in  parentheses. 


We  found  no  statistically  significant  association  between  acquisition  regime  and 
average  annual  cancellation  rates.  The  average  annual  cancellation  rate,  however,  is 
significantly  higher  in  periods  when  acquisition  funding  is  declining  than  it  is  in  periods 
of  stable  or  increasing  funding.  Loosely  stated,  the  conclusions  are  that  cancellations  tend 
to  cluster  in  periods  when  the  DoD  budget  is  falling,  and  changes  in  acquisition  policy 
and  process  made  over  the  years  have  not  influenced  average  annual  cancellation  rates. 
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Conclusions 


The  conclusions  offered  here  include  both  generalities  and  exceptions  to  the 
generalities,  and  these  may  be  equally  important. 

Average  cohort  cancellation  rates  are  not  strongly  influenced  by  either  acquisition 
regime  or  funding  climate.  The  exceptions  to  this  rule  are  provided  by  the  Post-Carlucci 
DSARC,  which  had  a: 

•  Significantly  higher  cohort  cancellation  rate  than  the  other  acquisition  regimes 
in  the  bust  period  FY  1987-FY  1989;  and 

•  Significantly  higher  average  cohort  cancellation  rate  in  the  bust  period  than  in 
the  boom  period. 

This  exception  deserves  further  exploration  as  it  may  provide  an  excellent  illustration  of  a 
mechanism  that  could  help  limit  the  number  of  new  starts  to  what  likely  future  budgets 
will  support. 

In  contrast,  annual  cancellation  rates,  although  not  strongly  influenced  by 
acquisition  regime,  are  strongly  associated  with  funding  climate.  A  possible  exception  to 
the  first  part  of  this  rule  is  provided  by  the  DAB  post  AR,  which  had  a  significantly 
higher  average  annual  cancellation  rate  than  other  acquisition  regimes  during  periods  of 
Stable  or  Increasing  funding.  There  were  no  exceptions  to  the  second  part  of  the  rule.  The 
complications  and  uncertainties  involved  with  the  conclusions  on  the  effects  of  changes 
in  acquisition  regimes  should  not  be  allowed  to  obscure  this  point.  Sharp  cuts  in  DoD 
funding  are  associated  with  high  annual  cancellation  rates:  about  60  percent  of  all 
cancellations  took  place  in  two  periods  of  large  cuts  in  acquisition  funding. 

The  conclusions  offered  by  this  paper  primarily  are  useful  because  they  direct 
attention  to  considerations  whose  centrality  has  not  been  recognized  in  discussions  of 
acquisition  reform.  Attention  typically  has  focused  on  acquisition  policy  and  process. 
While  changes  in  acquisition  policy  and  process  are  quite  possibly  justified  on  other 
grounds,  the  many  that  have  been  made  in  the  past  seem  to  have  influenced  cancellations 
at  most  episodically.  In  contrast,  changes  in  funding  climate  are  strongly  associated  with 
cancellations.  They  arguably  have  been  crucial.  DoD  force  structure,  the  capabilities  that 
the  Department  was  expected  to  provide,  and  funding  were  inconsistent  during  the  1970s 
and  for  more  than  a  decade  after  the  end  of  the  Cold  War.  That  inconsistency,  rather  than 
flaws  in  acquisition  policy  or  process,  is  the  context  in  which  most  cancellations  arise  and 
is  presumably  a  major  factor  to  be  considered  in  designing  proposals  for  improved 
outcomes. 
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A.  Introduction 


This  paper  is  motivated  by  a  2014  Institute  for  Defense  Analyses  (IDA)  paper1 
(hereafter  McN-W  (2014))  on  whether  there  is  a  statistically  significant  association 
between  cost  growth  and  changes  over  time  in  acquisition  policies  and  processes,  after 
taking  into  account  funding  climate.  McN-W  (2014)  found  that  there  is  not  a  statistically 
significant  association  between  cost  growth  and  the  changes  in  acquisition  policy  and 
process  that  had  occurred.  There  was,  however,  a  statistically  significant  association 
between  cost  growth  and  funding  climate.  In  particular,  average  cost  growth  was  much 
higher  for  Major  Defense  Acquisition  Programs  (MDAPs)  that  entered  what  is  now 
called  System  Design  and  Development  (SDD)“  during  “bust”  phases  of  the  “boom-bust” 
cycles  in  Department  of  Defense  (DoD)  funding. 

McN-W  (2014)  used  cost  growth  data  only  for  programs  that  had  successfully 
completed  development  and  gone  into  production.  This  paper  is  an  extension  of  McN-W 
(2014)  to  MDAPs  that  were  cancelled.  The  program  outcome  metric  used,  however,  is 
cancellations,  not  cost  growth. 

The  next  two  sections  function  as  scene  setters.  Section  B  offers  a  brief  survey  of 
cancellations  and  truncations  of  MDAPs,  which  is  not  available  in  the  literature.  Section 
C  asks  whether  very  high  cost  growth  is  the  main  cause  of  most  cancellations.  For  the 
subset  of  programs  examined,  it  appears  that  it  is  not,  which  opens  the  door  to  a  deeper 
look  at  the  data. 

Section  D  describes  the  plan  of  the  analysis,  which  is  built  around  two  different 
measures  of  the  cancellations: 

•  The  cancellation  rate  for  the  cohort  of  MDAPs  that  passed  Milestone  (MS)  II/B3 
during  a  period  of  years.  For  example,  during  Fiscal  Year  (FY)  2001-FY  2002, 

15  MDAPs  passed  MS  B,  of  which  four  were  eventually  cancelled,  for  a  cohort 
cancellation  rate  of  27  percent.  During  FY  1990-FY  1993,  16  programs  passed 
MS  B,  of  which  only  one  was  eventually  cancelled,  for  an  average  cohort 
cancellation  rate  of  6  percent. 
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SDD  was  called  Full  Scale  Development  into  the  late  1980s  and  then  Engineering  and  Manufacturing 
Development. 

3  DoDI  5000.2,  issued  October  23,  2000,  formally  established  MSs  A,  B,  and  C  (in  place  of  MSs  I,  II, 
and  III)  as  the  main  decision  points  for  an  MDAP.  The  definitions  are  such  that  MS  B  is  placed  several 
months  earlier  in  the  process  than  MS  II. 
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•  The  average  number  of  cancellations  over  a  span  of  years.  For  example,  during 
the  four-year  period  FY  1986-FY  1989,  nine  MDAPs  were  cancelled,  for  an 
average  cancellation  rate  of  2.3/yr.  Two  MDAPs  were  cancelled  during  the  five 
years  FY  1994-FY  1998,  for  an  average  cancellation  rate  of  0.4/yr. 

Cohort  and  annual  cancellation  rates  are  examined  in  Sections  E  and  F,  respectively.  In 
each  of  these  sections,  the  question  asked  is  whether  the  acquisition  policies  and 
processes  in  effect  at  the  time  and  the  funding  climate  have  a  statistically  discernible 
association  with  cancellation  rates.  Are  cancellation  rates  particularly  low  for  MDAPs 
that  passed  MS  II/B  in  certain  acquisition  regimes?  Do  programs  that  pass  MS  II/B  when 
funding  is  especially  tight  have  a  significantly  higher  (or  lower)  cancellation  rate? 

B.  A  Brief  Census  of  Cancellations  and  Truncations 

This  paper’s  database  took  as  its  point  of  departure  that  of  McN-W  (2014).  Their 
database  was  intended  to  be  comprised  of  all  MDAPs  that  passed  MS  II/B  during  the 
years  1970-2007.4  It  included  309  programs.  Of  these,  sixty-one  had  been  identified  as 
cancelled,  thirteen  as  truncated,  and  three  as  reorganized,  for  a  total  of  seventy-seven 
programs.  We  also  considered  a  RAND  study  from  1996  that  identified  forty-three 
cancelled  programs.5  Of  these,  thirty-two  were  also  on  our  list  of  cancelled  programs; 
there  were  eleven  programs  on  RAND’s  list  not  on  ours,  and  we  had  identified  eleven 
programs  as  cancelled  that  were  not  on  RAND’s  list.  As  a  result,  our  list  of  suspected 
cancellations  increased  to  ninety-nine.  We  reviewed  each  of  these  ninety-nine  programs 
to  verify  that  it  was  correctly  classified  and  to  extract  additional  data  from  the  Selected 
Acquisition  Reports  (SARs)  for  the  cancelled  programs.  The  compact  disc  (CD)  in  the 
pocket  on  the  inside  back  cover  of  this  paper  provides  brief  notes  on  ninety-two  of  these 
programs  that  were  problematical  in  one  way  or  another. 

1.  Definitions,  Criteria,  and  Issues  in  their  Application 

With  few  exceptions  the  differences  between  the  IDA  and  RAND  lists  of  cancelled 
programs  reflected  the  use  of  different  business  rules.  Reconciling  our  list  of  cancelled 
programs  with  RAND’s,  therefore,  required  an  explicit  criterion  for  cancellation  and 


The  database  includes  programs  that  passed  MS  II/B  at  the  Service  level  and  subsequently  fded  at  least 
one  SAR.  Not  all  of  these  were  Acquisition  Category  (ACAT)  I  programs.  The  database  also  includes  a 
small  number  of  programs  that  entered  at  MS  III/C  or  with  MS  II/B  authority  and  authorization  to 
proceed  to  Low  Rate  Initial  Production  (LRIP)  without  an  additional  authorization  from  the  Milestone 
Decision  Authority  (MDA). 

Jeanne  M.  Jarvaise,  Jeffrey  A.  Drezner,  and  Dan  Norton,  “The  Defense  System  Cost  Performance 
Database:  Cost  Growth  Using  Selected  Acquisition  Reports,”  RAND  Report  MR-625-OSD  (Santa 
Monica,  CA:  The  RAND  Corporation,  1996),  Appendix  B.  The  most  recent  SARs  used  in  the  RAND 
study  were  those  for  December  1994. 
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criteria  for  decisions  on  what  programs  should  be  in  the  population.  We  decided  to 
classify  a  program  as  cancelled  if: 

•  The  program  did  not  result  in  production  of  any  fully  configured  end  items,  or 

•  Any  fully  configured  end  items  produced  were  used  only  for  testing  and 
development. 

The  criteria  adopted  for  inclusion  in  the  population  of  programs  were  dictated  by  the 
purpose  of  the  analysis,  which  is  to  gauge  the  effect  of  different  Office  of  the  Secretary  of 
Defense  (OSD)-level  acquisition  regimes  and  funding  climates  on  cancellations.  Our 
population  then  should  include  only  programs  subject  to  OSD-level  acquisition  policy 
and  process.  Even  if  a  program  files  one  or  more  SARs,  OSD  oversight  pre-MS  II/B 
generally  is  less  rigorous  than  it  is  post-MS  II/B,  and  ACAT  II,  III,  and  IV  programs 
generally  are  not  subject  to  OSD-level  oversight.  Finally,  once  a  program  loses  ACAT  I 
status,  it  typically  reverts  to  Service-level  oversight  and  cannot  be  tracked  with  the  data 
sources  available  for  this  paper.  Accordingly,  we  exclude  programs  cancelled 

•  before  they  passed  MS  II/B; 

•  before  they  were  designated  as  an  ACAT  I  program;  or 

•  after  they  were  removed  from  ACAT  I  status  because  they  no  longer  met  the 
relevant  criteria. 

A  program  that  met  any  of  these  three  conditions  was  not  only  excluded  from  the  list  of 
cancelled  programs  but  from  the  population.  Programs  that  entered  SDD  before  the 
establishment  of  the  Defense  Systems  Acquisition  Review  Council  (DSARC)  in  1969 
also  were  excluded  from  the  population,  as  were  Special  Access  programs.  Finally,  the 
population  did  not  include  any  programs  that  passed  MS  B  after  FY  2007. 6  These 
exclusions  reduced  the  number  of  programs  included  in  the  main  database  from  309  to 
278. 7  The  main  database  is  the  first  Excel  file  on  the  included  CD. 

Application  of  these  criteria  was  not  entirely  cut  and  dried  for  three  sets  of 
programs.  First,  we  identified  six  programs  that  passed  MS  II/B  at  the  Service  level,  later 
filed  SARs,  and  subsequently  were  cancelled.  We  retained  on  the  list  of  cancelled 
programs  the  five  that  had  been  formally  designated  as  an  ACAT  I  program  and  excluded 


In  order  to  allow  a  reasonable  period  for  cost  growth  to  appear,  the  McN-W  (2014)  database  included 
only  programs  that  were  at  least  five  years  into  SDD.  The  December  2012  SARs  were  the  most  recent 
available  when  that  database  was  compiled,  so  the  last  cohort  included  in  the  population  was  the  one 
that  passed  MS  B  in  FY  2007.  The  database  for  the  present  paper  includes  cancellations  of  those 
programs  through  the  end  of  FY  2012. 

The  A-6E  and  the  A-6F  were  reported  on  the  same  SAR.  The  A-6E  was  a  successful  program,  while  the 
A-6F  was  cancelled.  For  that  reason,  the  A-6E  and  the  A-6F  are  entered  separately  on  the  list  of 
programs. 
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the  one  that  had  not.  Second,  there  is  a  question  as  to  whether  three  systems  from  about 
1969  actually  passed  a  formal  MS  II  review.  These  are  the  B-1A  Lancer,  the  XM800 
Armored  Reconnaissance  Vehicle  (SCOUT),  and  the  XM803  Main  Battle  Tank.  The 
DSARC  had  been  established  a  very  short  time  when  these  programs  entered  SDD.  There 
was  evidence  of  OSD  involvement  in  the  launch  of  these  programs,  however,  and  on  that 
basis  they  were  included  in  the  population  and  on  the  list  of  cancelled  programs.  Finally, 
as  noted  above,  the  population  includes  a  small  number  of  programs  that  entered  at  MS 
III/C  or  with  MS  II/B  authority  and  authorization  to  proceed  to  LRIP  without  an 
additional  authorization  from  the  MDA. 

We  classified  as  cancelled  fifty-two  of  the  MDAPs  in  the  population  using  the 
criteria  stated  above.  The  C-27J  also  was  included  on  the  list  of  cancelled  programs,  for  a 
total  of  fifty-three,  because  the  twenty-one  C-27Js  produced  were  placed  directly  in  long¬ 
term  storage.  The  second  Excel  file  on  the  included  CD  lists  these  programs  and  provides 
some  information  on  each  of  them. 

We  identified  sixteen  additional  programs  that  filed  one  or  more  SARs  during 
FY  1970-FY  2007  and  were  cancelled.  These  programs  are  listed  in  the  “Notes  on 
Programs  Examined”  file  provided  on  the  CD  included  with  this  paper;  they  appear  as 
numbers  54-69.  These  sixteen  were  not  included  on  the  list  of  cancelled  programs 
because  they  were  either  cancelled  before  passing  MS  II/B,  were  never  designated  an 
ACAT  I  program,  or  were  cancelled  after  they  fell  below  the  ACAT  I  level. 

2.  Cancellations 

Just  under  one  in  five  of  the  278  MDAPs  of  our  population  had  been  cancelled  by 
the  end  of  FY  2012  (see  Table  1).  Consistent  with  the  conventional  wisdom,  the 
cancellation  rate  for  joint  programs  (27  percent)  is  somewhat  higher  than  the  average  for 
Service-managed  programs  (about  18  percent).  The  Department  of  the  Navy  has  the 
lowest  cancellation  rate  (15  percent)  and  the  Army  the  highest  (20  percent).  The 
differences  are  not  large,  however.  Had  the  Navy  cancelled  two  more  programs  (over  the 
course  of  43  years),  the  Army  two  less,  and  the  Air  Force  one  less,  the  Services  would 
have  had  virtually  the  same  cancellation  rates. 


An/WQR-Advanced  Deployable  System,  AQM-127A  Supersonic  Low  Altitude  Target,  Advanced  Seal 
Delivery  System,  ASM-135A  Air-Launched  Anti-Satellite  System,  and  Land  Warrior.  Extended  Range 
Munition  was  cancelled  before  it  was  designated  an  ACAT  I  program. 


4 


Table  1.  Cancellations,  Total  Programs,  and  Cancellation  Rate 
by  Military  Department  and  Joint  Programs 


No.  of  Cancellations 

No.  of  Programs 

Cancellation  Rate 

Army 

14 

69 

20% 

Navy 

14 

96 

15% 

Air  Force 

13 

68 

19% 

Joint 

12 

45 

27% 

Total 

53 

278 

19% 

The  final  SAR  for  a  program  that  has  been  cancelled  usually  identifies  (with 
varying  degrees  of  clarity)  who  initiated  the  cancellation.  For  example:  “President  Bush 
ordered  the  termination  of  [SRAM  II]  on  27  Sept.  [  1 9]9 1  .”9  An  example  from  the  “less 
clear”  end  of  the  scale  is  provided  by  the  Joint  Ground  Launched  Tacit  Rainbow.  The 
final  SAR  for  the  program  states  that  its  funding  was  not  included  in  the  “FY92-FY93 
President’s  Budget”  and  that  the  program  was  cancelled  by  the  Secretary  of  Defense.10 
Using  such  statements  from  the  SARs  supplemented  by  materials  found  on  limited 
searches  on  the  internet,  the  initiative  for  the  cancellation  for  each  cancelled  program  was 
assigned  to  the  Service,  OSD,  the  White  House,  or  the  Congress.  In  those  cases  in  which 
the  information  available  left  substantial  doubt,  responsibility  for  initiating  the 
cancellation  was  assigned  to  OSD. 

There  is  a  limitation  of  the  attributions  made  in  this  way  that  needs  to  be  noted.  A 
cancellation  may  actually  be  initiated  by  an  organization  other  than  the  one  formally 
responsible  for  it.11  There  are  cases  in  which,  for  example,  a  Service  cancels  a  program 
because  it  appears  to  be  highly  likely  that  if  it  does  not  do  so,  OSD  or  the  Congress 
will — in  which  case,  OSD  or  the  Congress  will  decide  how  to  reallocate  the  funding.  In 
other  cases  the  cancellation  may  result  from  interactions  of  some  combination  of  the 
Military  Departments,  OSD,  the  White  House,  and  the  Congress.  None  of  these  sorts  of 
dynamics  was  discernible  in  the  information  consulted  in  developing  the  assignments  of 
responsibility  for  initiating  the  cancellation. 

Table  2  reports  the  number  of  cancellations  formally  initiated  by  OSD,  the  White 
House,  and  the  Congress.  Nothing  in  this  table  seems  particularly  surprising,  but  it  is 
worth  noting  that  about  34  percent  of  cancellations  were  initiated  by  the  Services. 


9  Selected  Acquisition  Report  (SAR)  for  SRAM  II,  December  3 1,  1991,  7. 

10  SAR  for  the  BGM-136  Joint  Ground  Launched  (JGL)  Tacit  Rainbow,  December  31,  1990,  4. 

1 1  MDAP  cancellations  are  visible  in  the  DoD  Budget  proposed  to  the  Congress  by  the  President,  and  in 
that  sense  all  cancellations  are  made  on  the  authority  of  the  President. 
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Table  2.  Number  and  Proportion  of  Cancellations  Initiated  by 
Different  Levels  of  Government 


Proportion  of 

No.  of  Cancellations  Cancellations 


Military  Departments 

18 

34% 

OSD 

23 

43% 

White  House 

4 

7% 

Congress 

8 

15% 

Total 

53 

100% 

Adds  to  less  than  100  percent  due  to  rounding. 


The  definition  of  “cancellation”  used  in  this  paper  ensures  that  all  cancelled 
programs  were  in  SDD  or  in  the  early  stages  of  LRIP.  In  fact,  all  but  eight  were  in  SDD 
and  only  one,  the  C-27J,  was  in  Full  Rate  Production.  Moreover,  the  average  time  from 
MS  II/B  to  cancellation  was  5.2  years,  and  half  of  all  cancellations  occurred  when  the 
program  was  no  more  than  4.3  years  beyond  MS  II/B,  as  shown  in  Figure  1.  However, 
the  distribution  has  a  fairly  long  tail.  One  program  was  cancelled  over  eighteen  years 
after  it  passed  MS  II  and  two  others  were  cancelled  after  more  than  twelve  years. 
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Figure  1.  Cumulative  Distribution  of  Time  between  MS  II/B  and  Cancellation 


It  would  be  desirable  to  distinguish  between  MDAPs  that  were  cancelled  because 
the  program  failed  in  some  way  and  programs  that  were  cancelled  because  external 
circumstances  changed.  Changes  in  the  threat,  for  example,  or  large  unanticipated 
reductions  in  funding,  or  emergence  of  new  technologies,  are  situations  in  which 
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cancellation  of  some  programs  may  be  the  best  course  of  action.  Section  D  identifies  a 
few  instances  of  programs  that  probably  were  in  large  part  cancelled  because  of  changes 
in  the  threat.  Unfortunately,  the  resources  available  for  this  paper  did  not  permit  an 
attempt  to  classify  all  cancellations  along  these  lines. 

3,  Truncations 

Severely  truncating  the  number  of  a  system  procured  is  likely  to  have  much  the 
same  consequence  as  cancellation  of  the  program.  Accordingly,  it  is  reasonable  to  extend 
the  census  to  truncations. 

The  database  used  in  this  paper  (and  found  on  the  included  CD)  contains  the 
MS  II/B  total  acquisition  quantity  (AQ)  and  the  quantity  actually  acquired  or,  for 
programs  still  underway,  the  planned  quantity  for  all  of  the  seventy-one  programs  that 
passed  MS  II/B  as  AC  AT  I  programs  during  FY  1989-FY  2007.  Figure  2  is  a  histogram 
of  the  percentage  of  the  respective  MS  II/B  AQs  acquired  by  these  seventy-one  MDAPs. 
Resource  limitations  precluded  extracting  from  the  SARs  the  corresponding  data  for 
programs  from  FY  1970-FY  1988. 


Figure  2.  Histogram  of  the  Percentage  of  the  MS  II/B  AQ  Acquired  by  MDAPs  that  Passed 
MS  II/B  as  ACAT  I  Programs,  FY  1989-FY  2007 


About  three-quarters  of  the  MDAPs  that  passed  MS  II/B  in  the  period  FY  1989- 
FY  2007  acquired  at  least  90  percent  of  their  MS  II/B  AQ.  The  median  program  acquired 
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100  percent  and  the  average  program  acquired  111  percent.  ~  These  figures  are  somewhat 
higher  than  is  commonly  appreciated  and  serve  to  limit  the  extent  to  which  truncations 
reasonably  can  be  viewed  as  a  major  problem.  If  the  experience  of  MDAPs  that  passed 
MS  II/B  during  FY  1989-FY  2007  is  representative,  the  great  majority  of  MDAPs  that 
were  not  cancelled  eventually  acquired  most  of  their  MS  II/B  AQ. 

Eighteen  of  the  seventy-one  MDAPs,  however,  acquired  less  than  90  percent  of 
their  AQs,  and  nine  acquired  less  than  half.  These  programs  and  the  percent  of  their 
respective  AQs  acquired  are  listed  in  Table  3.  ATIRCM  (second  entry  in  the  left  column 
of  the  table)  would  fit  comfortably  on  the  list  of  cancelled  programs.  The  same  cannot  be 
said  without  a  more  detailed  examination  of  any  of  the  other  seventeen  programs  that 
acquired  less  than  90  percent  of  their  MS  II/B  AQ,  and  at  least  one  (the  F/A-18E/F)  is 
generally  regarded  as  a  model  program. 


Table  3.  MDAPs  that  Passed  MS  II/B  as  ACAT  I  Programs  FY  1989-FY  2007  that  Procured 
at  Most  90%  of  their  MS  II/B  AQ  as  of  the  December  31,  2012  SAR 


MDAP 

MS 

II/B 

%  MS  II/B 
AQ 

MDAP 

MS 

II/B 

%  MS  II/B 
AQ 

Excalibur 

1997 

10 

F/A-18E/F 

1992 

55 

ATIRCM 

1995 

10 

B-1 B  CMUP 

1995 

65 

F-22 

1991 

29 

SBIRS  High 

1997 

80 

DDG-1000 

2006 

30 

AEHF  Satellite 

2001 

80 

AIM-9X  Sidewinder 

1996 

31 

JTRS  HMS 

2004 

82 

FGM-148A  Javelin 

1989 

36 

SMART-T 

1992 

83 

JSOW 

1992 

38 

AGM-158  JASSM  AAWS-M 

1998 

83 

C-5  RERP 

2001 

41 

EELV  (Atlas  V  &  Delta  IV) 

1998 

84 

HIMARS 

1999 

43 

AFATDS 

1989 

87 

Because  of  this  uncertainty,  the  analysis  presented  in  the  following  sections  is 
concerned  only  with  cancellations.  This  decision  was  reinforced  by  another 
consideration.  A  program  may  be  truncated  for  the  same  set  of  reasons  that  cause 
cancellations — shortcomings  of  the  program,  threat  changes,  funding  reductions,  and 
emergence  of  new  technologies.  In  addition,  a  truncation  can  be  caused  by  a  force 
structure  change.  A  careful  development  of  the  AQ  for  a  program  involves  a  range  of 
distinct  considerations — the  size  of  the  force  to  be  equipped  with  the  system,  the 
reliability  of  the  system,  policies  on  depot  maintenance  and  sparing,  war  reserve  stocks, 


This  number  excludes  the  observations  for  four  programs,  each  of  which  apparently  bought  more  than 
500  percent  of  its  MS  II/B  AQ.  These  figures  are  so  high  that  they  call  into  question  the  realism  of  what 
is  reported  in  the  SAR  as  the  MS  II/B  AQ. 
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and  basing.  A  change  in  any  of  these  could  produce  a  reduction  in  the  number  of  a  system 
needed. 


C.  Cost  Growth  and  Cancellations 

It  seems  to  be  the  conventional  wisdom  that  cost  growth  is  a  major  cause  of  most 
program  cancellations.  There  are  some  well-known  instances  in  which  this  was  the  case. 
The  literature,  however,  does  not  establish  that  it  is  a  typical  cause  of  program 
cancellations.  That  is  a  point  that  needs  to  be  checked  and  at  least  to  some  extent  can  be. 

Following  McN-W  (2014),  this  paper  uses  Program  Acquisition  Unit  Cost  (PAUC) 
as  its  measure  of  cost.  Cost  growth  is  computed  by  comparing  the  PAUC  of  the  MS  II/B 
baseline  with  the  PAUC  reported  in  a  later  SAR,  normalized  to  the  MS  II/B  baseline  AQ. 
The  PAUC  from  the  later  SAR  (in  the  same  base  year  dollars  as  the  MS  II/B  PAUC)  is 
the  actual  cost  or,  for  ongoing  programs,  an  estimate  based  on  significant  actual  cost 
experience. 

There  is  a  problem  with  computing  cost  growth  for  a  program  that  has  been 
cancelled:  the  final  SAR  rarely  reports  the  data  required  to  compute  quantity  adjusted 
PAUC  growth.  ~  It  is,  however,  possible  to  get  some  sense  of  the  amount  of  cost  growth 
that  occurred  prior  to  cancellation  by  going  back  to  the  most  recent  SAR  that  reported  the 
estimated  costs  for  developing  the  system  and  procuring  some  quantity  of  it.  Proceeding 
in  this  way,  we  were  able  to  estimate  cost  growth  for  twenty-five  of  the  fifty-three 
MDAPs  that  were  cancelled.  The  computations  are  documented  in  the  third  Excel  file  on 
the  included  CD.  The  SARs  did  not  report  the  data  needed  to  estimate  PAUC  growth  for 
the  other  twenty-eight  cancelled  programs. 

The  PAUC  growth  estimates  probably  understate  the  true  PAUC  growth,  for  two 
reasons.  First,  they  do  not  capture  the  cost  growth  between  the  date  of  the  SAR  used  and 
the  termination  of  the  program.  Second,  there  tends  to  be  some  delay  in  reporting  cost 
that  can  be  expected  to  occur  based  on  the  evidence  to  date  but  which  has  not  yet  in  fact 
materialized.  This  might  be  especially  the  case  for  programs  that  were  cancelled. 

Table  4  presents  a  comparison  of  PAUC  growth  data  for  the  151  MDAPs  used  in 
McN-W  (2014)  (none  of  which  was  cancelled)  and  PAUC  growth  for  twenty-five 
MDAPs  that  were  cancelled.  The  average  PAUC  growth  and  the  proportion  of  the  sample 
with  a  PAUC  growth  of  at  least  50  percent  are  both  higher  for  programs  that  were 
cancelled,  although  neither  difference  is  statistically  significant  using  conventional 


It  reports  the  Research,  Development,  Test  and  Evaluation  and  any  Procurement  funding  that  will 
actually  have  been  expended  when  the  program  is  over.  Usually  these  expenditures  do  not  result  in  the 
production  of  any  fully  configured  end  items,  and  the  SAR  does  not  report  what  it  would  cost  to 
complete  the  development  program  and  procure  some  quantity  of  the  system. 
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statistical  tests  and  a  significance  level  of  5  percent.14  The  proportion  of  cancelled 
MDAPs  that  (pre-cancellation)  had  a  negative  PAUC  growth  is  very  slightly  above  that 
for  programs  that  were  not  cancelled,  but  again  the  difference  is  not  statistically 
significant.15  In  short,  based  on  the  twenty-five  programs  examined,  programs  that  were 
cancelled  did  have  higher  cost  growth  on  average  than  programs  that  were  not  cancelled, 
but  not  greatly  higher,  and  fourteen  of  the  twenty-five  programs  would  not  even  have  had 
a  significant  Nunn-McCurdy  breach  after  nonnalizing  for  quantity. 

Table  4.  Characteristics  of  Quantity  Normalized  PAUC  Growth,  Samples  of  Not  Cancelled 

and  Cancelled  MDAPs 


Not  Cancelled  Cancelled 

Average  PAUC  Growth  32%  (151)  42%  (25) 

Distribution  of  PAUC  Growth 

>50%  26%  (40)  40%  (10) 

>30%  but  <  50%  1 1  %  (1 7)  0%  (0) 

0  to  <30%  46%  (64)  40%  (10) 

<0  19%  (30)  20%  (5) 


Note:  Numbers  in  parentheses  are  the  number  of  MDAPs. 

Fifty  of  the  176  MDAPs  for  which  we  have  a  PAUC  growth  estimate  showed 
growth  of  at  least  50  percent.  (See  the  third  row  of  Table  4.)  Of  these,  ten  were  cancelled 
and  forty  were  not.  The  sample  cancellation  rate  for  programs  with  PAUC  growth  of  at 
least  50  percent  is  therefore  20  percent.  The  cancellation  rate  for  programs  with  cost 
growth  of  less  than  50  percent  is  much  less — about  12  percent.  The  key  point  for  present 
purposes,  however,  is  that  80  percent  of  the  MDAPs  with  cost  estimates  of  at  least  50 
percent  were  not  cancelled.  In  short,  there  is  more  than  cost  growth  influencing  MDAP 
cancellations.16 


A  t-test  indicates  there  is  no  statistical  difference  between  the  mean  PAUC  growth  rates  for  cancelled 
and  non-cancelled  programs.  Two-sample  tests  of  proportions  indicate  there  is  no  statistical  difference 
in  the  proportion  of  programs  at  certain  PAUC  growth  rate  thresholds  (>  50  percent,  <  30  percent,  and 
<  0)  between  the  sample  of  cancelled  and  non-cancelled  programs. 

It  is  also  worth  noting  that  PAUC  growth  for  programs  that  were  cancelled  shows  the  same  pattern  with 
respect  to  funding  climate  as  does  average  PAUC  growth  of  programs  that  were  not  cancelled.  That  is, 
MDAPs  that  passed  MS  II/B  during  bust  phases  of  the  boom-bust  cycle  had  higher  average  PAUC 
growth,  a  greater  portion  with  PAUC  growth  of  at  least  50  percent,  and  a  smaller  portion  with  a 
negative  PAUC  growth.  There  statements  for  the  cancelled  programs  can  be  verified  using  the  third 
Excel  workbook  referred  to  above. 

Given  the  estimates  in  Table  7,  the  maximum  number  of  programs  cancelled  that  had  cost  growth  of  at 
least  50  percent  is  38,  which  would  imply  that  just  over  50  percent  of  programs  with  cost  growth  of  at 
least  50  percent  were  not  cancelled. 


10 


D.  Plan  of  the  Analysis 

Those  in  the  defense  community  are  likely  to  reach  for  “other  factors”  relevant  to 
cancellation  in  either  of  two  boxes.  One  of  these  is  acquisition  policy  and  process,  which 
is  presumed  to  influence  MDAPs  in  ways  important  to  their  success  or  lack  thereof — for 
example,  contract  type,  maturity  of  key  technologies  to  be  used,  realism  of  the  cost 
estimate,  and  the  degree  of  concurrence  between  SDD  and  production,  among  others. 
Within  this  context,  the  relevant  measure  is  cancellation  rates  for  cohorts  of  MDAPs  that 
passed  MS  II/B  during  different  periods.  We  wish  to  know  whether  cohort  cancellation 
rates  were  significantly  higher  or  lower  under  some  of  these  regimes  than  others. 

The  other  box  is  changes  in  funding.  When  DoD  funding  declines  substantially,  the 
Department  must  make  cuts  in  programs  and  it  would  be  no  surprise  to  find  that  includes 
cancellation  of  some  MDAPs.  In  this  context,  the  relevant  measure  is  cancellations 
organized  by  the  year  in  which  they  occur,  regardless  of  when  the  cancelled  programs 
passed  MS  II/B. 

To  use  these  points  as  the  frame  for  an  analysis,  it  is  necessary  to  specify  acquisition 
regimes  and  funding  climates.  This  paper  distinguishes  five  successive  DoD  acquisition 
regimes:17 

1.  Defense  Systems  Acquisition  Review  Council  (DSARC),  1970-1982 

2.  Post-Carlucci  Initiatives  DSARC,  1983— 198918 

3.  Defense  Acquisition  Board  (DAB),  1990-1993 

4.  Acquisition  Reform  (AR),  1994-2000 

5.  DAB  post  AR,  2001-2012 

For  reasons  that  are  brought  out  below,  the  cohort  cancellation  rate  requires  a 
different  specification  of  breakpoints  between  funding  climates  than  does  the  annual 
cancellation  rate.  In  broad  terms,  however,  there  were  two  complete  boom-bust  cycles 
during  the  period  1970-2008: 

•  The  reduction  in  DoD  funding  associated  with  the  winding  down  of  US 
involvement  in  the  War  in  Vietnam  through  the  end  of  the  Carter-Reagan 
buildup,  roughly  1970-1986. 


The  main  reference  we  have  used  is  J.  Ronald  Fox,  Defense  Acquisition  Reform,  1960  to  2009:  An 
Elusive  Goal  (Washington,  DC:  US  Army  Center  of  Military  Flistory,  2011).  Fox  identifies  the  key 
features  of  the  first  four  of  these  periods. 

After  then-Deputy  Secretary  of  Defense  Frank  Carlucci.  There  is  some  uncertainty  about  when  the  Post- 
Carlucci  Reforms  DSARC  should  end  and  the  DAB  regime  should  begin.  The  relevant  statutes  were 
passed  in  1986,  and  the  DAB  began  functioning  under  that  name  in  late  FY  1987  or  early  FY  1988; 
however,  DoD  did  not  implement  the  full  set  of  reforms  required  by  statute  until  1990.  We  have  for  that 
reason  set  the  line  at  1990. 
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•  The  post-Cold  War  reduction  in  DoD  funding  and  the  subsequent  9-11  buildup, 
roughly  1987-2008. 

The  start  of  the  withdrawal  of  US  forces  from  Iraq  in  2009  marked  the  start  of  a  third 
cycle,  which  was  ratified  by  passage  of  the  Budget  Control  Act  (BCA)  in  201 1. 

Changes  over  time  in  acquisition  regime  and  the  boom-bust  cycle  in  DoD  funding 
provide  natural  experiments  that  may  shed  some  light  on  the  effects  of  acquisition  policy 
and  funding  climate  on  cancellation  rates.  For  example,  each  of  the  five  acquisition 
regimes  was  in  place  during  a  bust  phase  in  the  cycle.  By  looking  at  changes  in  the  cohort 
cancellation  rate  from  one  acquisition  regime  to  the  next  during  these  periods,  we  may  be 
able  to  see  effects  of  changes  in  acquisition  policy  or  process.  Similarly,  three  of  the 
acquisition  regimes  were  in  effect  during  both  a  bust  phase  and  a  boom  climate,  and  so 
may  provide  some  insight  into  the  effect  of  a  change  in  funding  climate  given  acquisition 
regime. 

An  analysis  of  natural  experiments  can  be  useful,  but  it  is  necessary  to  recognize, 
that  such  experiments  tend  to  be  poorly  controlled.  That  is,  in  addition  to  the  easily 
identified  elements  of  the  natural  experimental  design,  there  are  other  factors  that  may 
have  influenced  cohort  cancellation  rates.  We  first  focus  on  acquisition  regime  and 
funding  climate  and  then,  after  statistical  results  for  the  basic  natural  experiments  are 
stated,  ask  whether  they  are  compromised  by  the  omission  of  other  factors. 

The  following  section  examines  whether  there  are  statistically  significant 
associations  between  cohort  cancellation  rates  and  acquisition  regime  and  funding 
climate.  Section  F  presents  a  parallel  analysis  of  annual  cancellation  rates. 

E.  Analysis  of  Cohort  Cancellation  Rates 

Funding  and  programmatic  decisions  embedded  in  the  MS  II/B  baselines  of  MDAPs 
are  made  at  least  one  or  two  years  before  the  President’s  Budgets  in  question  are 
submitted  to  the  Congress.  Consequently,  those  decisions  necessarily  reflect  expectations 
held  by  decision  makers  about  the  future  DoD  budget  climate,  and  the  breakpoints 
between  different  budget  climates  should  mark  the  points  at  which  there  were  major 
shifts  in  expectations.  We  used  three  events  to  identify  the  breakpoints  between  funding 
climates:  (1)  the  invasion  of  Afghanistan  by  the  USSR  in  late  December  1979; 
(2)  passage  of  the  Gramm-Rudman-Hollings  (GRH)  Act  in  December  1985;  and  (3)  the 
terrorist  attack  on  the  United  States  on  September  11,  2001.  Analysis  of  surrounding 
presidential  statements  and  budgetary  events  led  to  the  following  breakpoints  between  the 
funding  climates: 

•  FY  1981  as  the  first  year  of  the  Carter-Reagan  buildup, 

•  FY  1986  as  the  final  year  of  the  Carter-Reagan  buildup,  and 
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•  FY  2003  as  the  first  year  of  the  post-9/1 1  defense  buildup. 

The  funding  climates  will  be  referred  to  as  Relatively  Constrained  (or  bust)  and 
Relatively  Accommodating  (or  boom).  Senior  decision  makers  could  reasonably  expect 
each  of  the  events  identified  to  signal  major  and  sustained  changes  in  the  defense  funding 
climate,  which  in  fact  they  did. 

1.  Statistical  Analysis 

Table  5  provides  the  data  on  cohort  size,  number  of  cancellations,  and  cohort 
cancellation  rates  arrayed  by  acquisition  regime  and  funding  climate.  Note  that  two  of  the 
regimes  (DAB  and  AR)  operated  only  in  bust  climates,  so  we  have  only  three  natural 
experiments  on  the  effect  of  a  change  in  funding  climate  for  a  given  acquisition  regime. 


Table  5.  Average  Cancellation  Rates  for  Cohorts,  by  Acquisition  Regime  and  Funding 

Climate 


Relatively  Constrained 

Relatively  Accommodating 

Acquisition  Regime 

Period 

Cancellation  Rate 

Period 

Cancellation  Rate 

DSARC 

1970-1980 

14% 

(10  of  74) 

1981-1982 

24% 

(4  of  17) 

Post-Carlucci  DSARC 

1987-1989 

42% 

(11  of  26) 

1983-1986 

12% 

(6  of  51) 

DAB 

1990-1993 

7% 

(1  of  15) 

AR 

1994-2000 

14% 

(6  of  43) 

DAB  post  AR 

2001-2002 

27% 

(4  of  15) 

2003-2007 

30% 

(11  of  37) 

Note:  Number  of  cancellations  and  cohort  size  in  parentheses. 


The  one  feature  of  Table  5  that  stands  out  is  the  much  higher  average  cohort 
cancellation  rate  in  the  bust  climate  (42  percent)  than  in  the  boom  years  (12  percent)  for 
the  Post-Carlucci  DSARC.  A  standard  test  (the  t-test)  found  this  difference  to  be 
statistically  significant.  The  differences  for  the  other  two  budget  climate  experiments 
were  not  statistically  significant.  Two  more  comprehensive  tests  produced  these  same 
results.19 


19 

The  first  of  these  was  a  logit  regression  using  the  DSARC  as  the  reference  acquisition  regime.  Dummy 
variables  marked  the  other  acquisition  regimes  and  slope  dummies  were  used  to  capture  the  effect  of  the 
different  funding  climates.  We  also  applied  Fisher’s  exact  test  to  a  contingency  table  for  each  of  the  two 
climates,  which  rejected  the  null  hypothesis  that  cohort  cancellation  rates  are  independent  of  the 
acquisition  regime  in  the  bust  climate.  Inspection  of  the  data  strongly  suggests  that  this  result  is  due  to 
the  very  high  average  cohort  cancellation  rate  for  FY  1987-FY  1989.  McN-W  (2014)  used  Analysis  of 
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The  statistical  analysis  did  not  find  any  significant  association  between  acquisition 
regimes  and  the  average  cohort  cancellation  rate  within  the  boom  budget  climate,  but  the 
Post-Carlucci  DSARC  was  significantly  higher  than  the  others  in  the  bust  climate. 

2.  Discussion 

We  start  with  the  question  of  whether  it  is  historically  plausible  to  attribute  the 
higher  average  cancellation  rate  of  MDAPs  that  passed  MS  II  during  the  latter  years  of 
the  Post-Carlucci  DSARC  (FY  1987-FY  1989)  to  a  deterioration  of  the  funding  climate. 

The  backdrop  to  this  question  is  the  lower  cancellation  rate  of  programs  that  passed 
MS  II  during  FY  1983-FY  1986.  This  result  is  intelligible  in  terms  of  the  changes  made 
by  the  Carlucci  Initiatives,  which  were,  among  other  things,  intended  to  tighten  up  the 
DSARC  process  and  to  help  ensure  that  DoD  did  not  start  more  programs  than  reasonably 
anticipated  funding  would  support.  In  particular,  major  system  new  starts  became  a  joint 
responsibility  of  the  Planning,  Programming,  and  Budgeting  System  (PPBS)  and  the 
acquisition  process.  The  vehicle  used  was  a  Major  Systems  New  Starts  issue  paper 
developed  during  the  summer  Program  Review.  At  least  by  intent,  the  new  process  would 
favor  a  relatively  low  cohort  cancellation  rate. 

This  was  achieved  during  the  boom  years  FY  1983-FY  1986,  but  the  cancellation 
rate  for  the  cohort  from  the  following  bust  years  (FY  1987-FY  1989)  was  three  and  one- 
half  times  as  high — 42  percent  versus  12  percent.  There  is,  however,  good  reason  to 
doubt  that  all  of  the  cancellations  were  largely  due  to  funding  decreases. 

The  FY  1987-FY  1989  period  came  just  before  the  end  of  the  Cold  War.20  Some  of 
the  MDAPs  that  passed  MS  II  during  these  years  were  later  cancelled  because,  in  the 
light  of  altered  perceptions  of  the  threats,  they  no  longer  seemed  to  be  of  enough  utility  to 
justify  their  cost.  While  the  SARs  do  not  spell  out  the  fact,  there  is  little  doubt  that 
changes  in  the  threat  were  a  major  factor  in  the  cancellations  of  SRAM  II,  the  Small 
ICBM,  Peacekeeper  Rail  Garrison,  and  possibly  some  other  programs  from  this  cohort. 
To  some  extent,  then,  the  high  cancellation  rate  of  the  FY  1987-FY  1989  cohort  is 
attributable  to  the  end  of  the  Cold  War  rather  than  to  the  prevailing  acquisition  regime 
and  funding  climate. 


Variance.  (ANOVA);  the  statistical  analysis  of  average  cancellation  rates  reported  in  Section  F  below 
also  does.  ANOVA  is  not  used  in  this  section  because  it  is  not  suited  to  analysis  of  categorical  data 
(such  as  cancellations).  ANOVA  could  be  used  to  analyze  variations  in  the  cancellation  rates  of  annual 
cohorts  if  the  number  of  programs  that  pass  MS  II/B  each  year  is  sufficiently  large,  but  we  judged  it  not 
to  be. 

The  Berlin  Wall  fell  on  November  9,  1989,  just  over  one  month  into  FY  1990.  The  Soviet  Union  was 
formally  dissolved  in  April  1991. 
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The  high  cancellation  rate  for  the  FY  1987-FY  1989  cohort  also  may  in  part  be 
attributable  to  a  tacit  change  in  the  Administration’s  policy  on  new  starts.  The  amount 
appropriated  for  DoD  acquisition  fell  by  about  30  percent  from  FY  1986  to  FY  1989,  and 
with  the  passage  of  GRH  and  developments  in  Eastern  Europe  there  was  no  reason  to 
expect  an  increase  over  the  then  foreseeable  future.  In  the  years  FY  1987-FY  1989, 
however,  on  average  8.3  programs  passed  MS  II  annually.  This  rate  was  well  above  the 
rate  for  periods  of  Relatively  Constrained  funding  (6.7/yr.  for  FY  1979-FY  1980,  and 
4/yr.  for  FY  1990-FY  1993).  The  relatively  high  new  start  rates  suggest  a  decreased 
emphasis  on  the  Carlucci  Initiatives  goal  of  not  starting  more  MDAPs  than  likely  future 
budgets  could  sustain.  In  fact,  nine  of  the  eleven  cancellations  from  the  new  starts  in  the 
FY  1987-FY  1989  cohort  occurred  during  FY  1990-FY  1993. 

The  summary  conclusion  reached  depends  on  the  weight  given  to  the  historical 
considerations.  If  these  are  discounted,  in  one  of  the  three  relevant  natural  experiments — 
but  not  the  other  two — a  Relatively  Constrained  funding  climate  was  significantly 
associated  with  a  higher  average  cohort  cancellation  rate.  If,  instead,  the  historical  factors 
are  heavily  weighted,  the  conclusion  would  be  that  neither  acquisition  regime  nor  funding 
climate  mattered  for  cohort  cancellation  rates.  The  difference  between  these  two  positions 
is  thin.  A  reasonable  conclusion,  and  the  one  we  suggest,  is  that  the  analysis  shows  that 
neither  acquisition  policy  and  process  changes  nor  changes  in  funding  climate  have  had 
much,  if  any,  effect  on  cohort  cancellation  rates. 

Embedded  in  this  discussion  is  a  question  that  has  not  been  raised  explicitly  so  far: 
Are  cohort  cancellation  rates  significantly  influenced  by  events  that  occur  after  MS  II/B, 
especially  changes  in  budget  climate?  The  analysis  of  Table  5  implicitly  rules  out  such 
effects  by  attributing  a  cohort  cancellation  rate  only  to  the  budget  climate  and  acquisition 
regime  in  effect  when  a  program  passed  MS  II/B.  The  next  subsection  examines  annual 
cancellation  rates,  without  regard  to  when  the  cancelled  programs  passed  MS  II/B.  Not 
surprisingly,  cancellations  prove  to  be  concentrated  in  periods  of  declining  funding.  In 
the  remainder  of  this  subsection  we  consider  whether  these  cancellations  or  a  relative 
absence  of  cancellations  were  concentrated  in  a  way  that  casts  “shadows”  back  on 
particular  MS  II/B  cohorts.  The  underlying  issue  is  whether  such  shadows  may  have 
masked  effects  on  cohort  cancellation  rates  of  changes  in  acquisition  regime  or  budget 
climate. 

Shadow  effects  presumably  would  be  most  likely  to  appear  in  cohorts  that  passed 
MS  II/B  during  the  few  years  before  the  start  of  a  boom  or  the  start  of  a  bust.  There  are 
four  such  periods  during  FY  1970-FY  2007.  Somewhat  arbitrarily  using  periods  of  five 
years  duration,  these  are: 

•  Before  booms:  FY  1976-FY  1980  and  FY  1998-FY  2002 

•  Before  busts:  FY  1 982-FY  1 986  and  FY  2003-FY  2007 
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If  future  funding  climates  do  cast  shadows,  we  would  expect  the  programs  that  passed 
MS  II/B  in  the  years  before  a  boom  began  to  have  a  relatively  low  cancellation  rate  and 
those  that  passed  MS  II/B  before  a  bust  began  to  have  a  relatively  high  cancellation 
rate.  These  expectations  were  not  met  in  the  1970s  and  1980s  pre-boom  and  pre-bust 
periods.  They  were,  however,  met  in  FY  1998-FY  2002,  which  had  a  below  average 
cancellation  rate  (16  percent)  and  FY  2003-FY  2007,  which  had  an  above  average  cohort 
cancellation  rate  of  30  percent. 

FY  1998-FY  2000  is  at  the  end  of  the  AR  regime  and  FY  2001-FY  2002  is  at  the 
start  of  the  DAB  post  AR  regime.  This  fact — that  the  relevant  time  period  crosses 
acquisition  regimes —  complicates  the  problem.  Furthermore,  adjustment  of  the  average 
cancellation  rate  for  the  FY  1998-FY  2002  cohort  would  require  identifying  programs 
that  might  have  been  cancelled  except  for  the  good  fortune  of  entering  SDD  in  the  years 
before  a  boom  in  funding.  There  is  no  plausible  way  to  do  this.  ~  It  is  possible  within  the 
limits  of  this  study  to  do  only  a  little  better  on  the  FY  2003-FY  2007  cohort.  Of  the 
eleven  programs  of  this  cohort  cancelled,  seven  were  cancelled  during  FY  2009- 
FY  2012.'  These  had  on  average  spent  about  56  months  in  SDD.  Some  of  these 
programs  may  have  been  cancelled  largely  for  budgetary  reasons,  but  considerably  more 
effort  would  be  required  to  verily  that  assumption. 

With  the  evidence  available,  the  overall  conclusion  on  the  effects  of  acquisition 
regime  and  funding  climate  on  cohort  cancellation  rates  that  we  offered  earlier  remains 
reasonable:  neither  acquisition  policy  and  process  changes  nor  changes  in  funding 
climate  have  had  much,  if  any,  effect  on  cohort  cancellation  rates.  A  more  complete  and 
rigorous  analysis  of  the  role  played  by  future  funding  climates  would  be  desirable, 
however. 


It  might,  instead,  be  that  the  programs  started  in  times  of  lean  funding  tended  to  be  on  average  of  a 
higher  priority  and  therefore  less  likely  to  be  cancelled.  In  fact,  the  average  cancellation  rate  for 
programs  that  passed  MS  II/B  in  bust  periods  (18.3  percent)  is  slightly  less  than  that  of  programs  that 
passed  MS  II/B  in  boom  periods  (20.2  percent). 

We  perhaps  can  see  a  budget  effect  in  the  durations  of  programs  from  the  FY  1998-FY  2002  cohort  that 
eventually  were  cancelled.  One  of  the  five  programs  in  this  cohort  to  be  cancelled  (the  RAH-66)  was 
cancelled  in  FY  2000.  The  other  four  were  cancelled  in  FY  2010  or  FY  201 1.  By  this  point,  the  four 
programs  cancelled  had  been  in  SDD  for  an  average  of  nearly  10  years,  more  than  twice  the  average 
duration  of  programs  that  were  cancelled.  (See  Figure  1  on  page  6.) 

Future  Combat  Systems  (FCS),  Patriot/MEADS  Combined  Aggregate  Program  (CAP), 
Transformational  Satellite  Communications  System  (TSAT),  Armed  Reconnaissance  Helicopter 
(ARH),  Joint  Land  Attack  Cruise  Missile  Defense  Elevated  Netted  Sensor  System  (JLENS),  VH-71, 
and  C-27J. 
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F.  Analysis  of  Annual  Cancellation  Rates 

This  section  presents  an  analysis  of  annual  cancellation  rates.  The  institutional 
context  for  analysis  of  annual  cancellation  rates  is  the  program/budget  process.  Officials 
from  OSD  and  Service  acquisition  organizations  are  involved  in  the  program/budget 
process  but  do  not  administer  it.  It  is  not  uncommon — and  in  fact  probably  is  typical — for 
cancellations  to  be  initiated  in  the  programming  or  budgeting  process  of  a  Service  or  in 
the  OSD-level  programming  and  budgeting  processes. 

1.  Statistical  Analysis 

The  acquisition  regimes  relevant  to  this  section  are  the  same  as  those  used  in  the 
previous  section.  The  breakpoints  between  the  funding  climates  need  to  be  changed, 
however.  As  was  noted  above,  funding  decisions  (and  decisions  contingent  on  funding)  at 
MS  II/B  must  be  made  on  the  basis  of  expectations  because  the  actual  DoD  funding 
levels  are  not  yet  known  when  the  decisions  are  made.  In  contrast,  budget-driven 
cancellation  decisions  are  not  likely  to  be  made  until  the  need  for  doing  so  becomes  clear. 
Consequently,  they  generally  will  reflect  the  most  recently  enacted  budget  or  the 
upcoming  President’s  Budget.  The  breakpoints  used  reflect  the  actual  amounts  of  Budget 
Authority  for  Procurement  (in  constant  dollars)  appropriated  by  the  Congress  in  the 
relevant  fiscal  years.  Because  the  breakpoints  are  not  the  same  as  those  of  the  budget 
climates  used  in  the  preceding  section,  those  initially  used  in  this  section  will  be  labeled 
Decreasing  Funding  and  Increasing  Funding. 

Table  6  presents  average  cancellation  rates  organized  by  funding  climate  (columns) 
and  acquisition  regime  (rows).  As  in  the  previous  section,  for  a  given  funding  climate,  we 
can  examine  whether  major  changes  in  acquisition  policy  and  process  are  associated  with 
statistically  significant  differences  in  cancellations.  We  also  can  look  at  the  effect  of  bust 
versus  boom  in  cancellations  for  a  given  acquisition  policy  and  process  regime. 
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Table  6.  Average  Number  of  Cancellations  per  Fiscal  Year  by  Acquisition  Regime 

and  Funding  Climate 


Decreasing  Funding 

Increasing  Funding 

Acquisition  Regime 

Period 

Cancellation  Rate 

Period 

Cancellation  Rate 

DSARC 

1970-1975 

0.3/yr. 

(2  in  6  yrs.) 

1976-1982 

0.6/yr. 

(4  in  7  yrs.) 

Post-Carlucci  DSARC 

1986-1989 

2.3/yr. 

(9  in  4  yrs.) 

1983-1985 

0.3/yr. 

(1  in  3  yrs.) 

DAB 

1990-1993 

3.3/yr. 

(13  in  4  yrs.) 

AR 

1994-1998 

0.4/yr. 

(2  in  5  yrs.) 

1999-2000 

0.5/yr. 

(1  in  2  yrs.) 

DAB  post  AR 

2009-2012 

2.8/yr. 

(11  in  4  yrs.) 

2001-2008 

1 ,3/yr. 

(10  in  8  yrs.) 

Note:  Number  of  cancellations  in  parentheses. 


The  statistical  analysis  divides  the  acquisition  regimes  into  two  groups:24 

•  Post-Carlucci  DSARC,  DAB,  and  DAB  post  AR.  In  neither  funding  climate 
are  these  regimes  significantly  different  from  each  other.  The  DAB  regime 
operated  only  in  a  bust  climate.  The  other  two  had  a  significantly  higher  average 
cancellation  rate  in  the  Decreasing  Funding  climate  than  in  the  Increasing 
Funding  climate.  For  this  group,  acquisition  regime  had  no  significant  effect  on 
the  average  number  of  MDAPs  cancelled  each  year,  but  the  cancellation  rates 
were  much  higher  in  periods  of  Decreasing  Funding. 

•  DSARC  and  AR.  These  regimes  also  do  not  differ  significantly  from  each  other 
but  have  comparatively  low  cancellation  rates  in  each  funding  climate.  Neither 
acquisition  regime  nor  funding  climate  matters  significantly  in  these  cases. 

Within  each  of  these  groups,  the  average  number  of  programs  cancelled  per  year  (for  a 
given  funding  climate)  is  similar,  but  only  in  the  first  group  do  we  see  a  marked 
difference  in  cancellation  rates  between  the  two  funding  climates.  This  feature  of  the 


Statistical  results  discussed  are  based  on  three  separate  tests:  Analysis  of  Variance  (ANOVA),  Tukey- 
Kramer,  and  t-tests.  ANOVA  is  used  to  determine  whether  there  is  a  statistical  difference  in  the  mean 
number  of  cancellations  across  the  different  acquisition  regimes  within  each  funding  climate.  The  null 
hypothesis  (no  statistical  difference  in  means  across  regimes)  is  only  rejected  for  the  decreasing  funding 
climate.  A  Tukey-Kramer  test  is  then  used  to  perform  pair-wise  comparisons  and  identify  which 
specific  regimes  are  statistically  different  from  one  another  during  the  decreasing  funding  climate. 
T-tests  test  the  difference  in  each  acquisition  regime’s  cancellation  rate  across  the  two  funding  climates. 
A  t-test  indicates  that  pooled  annual  cancellation  rates  are  statistically  different  across  the  decreasing 
and  increasing  funding  climates  (p-value  =.063).  When  regime-specific  t-tests  are  performed,  the  Post- 
Carlucci  DSARC  and  DAB  post  AR  are  the  only  regimes  found  to  have  cancellation  rates  that  differ 
statistically  across  funding  climates. 
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results  suggests  that  one  or  more  factors  not  yet  identified  affect  the  outcome  of  our 
natural  experiments. 

2.  Discussion 

Figure  3  shows  the  number  of  cancellations  in  each  fiscal  year.  Two  distinct  clusters 
of  cancellations  are  evident  in  this  figure:  FY  1986-FY  1992  and  FY  2009-FY  2012. 
These  were  both  periods  of  steep  decline  in  procurement  funding. 

FY  1986-FY  1993  was  the  bust  phase  following  the  Carter-Reagan  boom  in  defense 
budgets.  In  December  1985,  President  Reagan  signed  into  law  the  GRH.  Like  the  BCA 
adopted  some  twenty-five  years  later,  GRFI  set  binding  budget  limits  and  imposed 
sequestration  on  appropriations  in  excess  of  those  limits.  From  peak  procurement  Budget 
Authority  of  $177.9  billion  in  FY  1985,  procurement  fell  to  $76.5  billion  in  FY  1993,  43 
percent  of  its  FY  1985  level.  Twenty-two  MDAPs  were  cancelled  during  FY  1986- 
FY  1993. 


Figure  3.  Number  of  MDAPs  Cancelled  Each  Year,  1970-2012 


The  second  period  of  declining  funding  was  FY  2009-FY  2012.  The  financial  crisis 
occurred  towards  the  end  of  Calendar  Year  (CY)  2008,  or  approximately  during  the  first 
quarter  of  FY  2009.  The  withdrawal  of  US  troops  from  Iraq  began  in  July  of  2009  and  the 


These  data,  as  well  as  similar  funding  data  cited  later,  are  in  billions  of  constant  FY  2014  dollars  of 
budget  authority.  They  are  from  Table  6-8  (p.  142ff)  of  the  Office  of  the  Under  Secretary  of  Defense 
(Comptroller),  National  Defense  Budget  Estimates  for  FY  2014. 
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BCA  was  signed  into  law  late  in  FY  2011.  FY  2012  DoD  procurement  funding  constant 
dollars  stood  at  about  two-thirds  of  its  FY  2008  level.  Eleven  MDAPs  were  cancelled 
during  FY  2009-FY  2012.  Thirty-three  MDAPs — over  60  percent  of  the  total — were 
cancelled  during  the  twelve  years  of  these  two  periods  taken  together,  an  average  of 
nearly  three  cancellations  each  year.  During  the  other  thirty-one  years  on  average  about 
two  MDAPS  were  cancelled  every  three  years  (for  a  total  of  20). 

The  absence  of  a  cluster  of  cancellations  during  the  early  to  mid-1970s  is 
particularly  surprising  because  this  apparently  was  the  bust  phase  of  a  boom-bust  cycle 
associated  with  US  involvement  in  the  War  in  Vietnam.  Procurement  funding  during  the 
Vietnam  War  in  constant  dollars  averaged  about  20  percent  above  its  prewar  level. 
Withdrawal  of  US  forces  from  Vietnam  began  in  1969.  Virtually  all  US  forces  had  been 
withdrawn  by  the  end  of  1972,  but  US  material  support  for  South  Vietnam  continued  into 
1975.  In  FY  1969,  procurement  funding  in  constant  dollars  was  $116.8  billion.  Real 
procurement  funding  was  down  by  15.5  percent  in  FY  1970,  and  by  FY  1975  it  was  just 
over  half  what  it  had  been  in  1969.  This  decline  was  less  than  that  following  the  Carter- 
Reagan  defense  buildup  but  greater  than  the  decline  of  FY  2009-FY  2012. 

A  likely  explanation  for  the  comparatively  low  cancellation  rate  during  FY  1970- 
FY  1975  can  be  found  in  changes  over  the  relevant  period  in  the  composition  of 
procurement  funding.  A  large  part  of  the  increase  in  procurement  funding  during 
FY  1962-FY  1969  was  for  munitions  and  procurement  to  replace  systems  lost  in  combat, 
particularly  aircraft.  These  increases  were  reversed  as  the  US  forces  were  withdrawn  and 
US  support  of  South  Vietnam  reduced  and,  in  1975,  ended.  Insofar  as  procurement  of 
MDAPs  is  concerned,  there  was  little  or  no  boom-bust  cycle  associated  with  the  Vietnam 
War.  ~  Except  for  FY  1970,  procurement  funding  in  constant  dollars  during  this  period 
remained  below  its  prewar  peak,  however. 

The  period  FY  1994-FY  1998  seems  to  have  a  similar  explanation.  Although 
procurement  funding  in  constant  dollars  was  17.7  percent  lower  in  FY  1994  than  it  was  in 
FY  1993,  across  the  four  fiscal  years  FY  1995-FY  1998,  funds  budgeted  for  procurement 
declined  only  very  slightly — from  $62.8  billion  to  $60.5  billion.  FY  1994-FY  1998  was 
then  a  period  of  comparatively  stable  procurement  funding. 

The  implication  of  these  comments  is  that  periods  of  sharply  decreasing  funding 
should  be  treated  as  a  separate  funding  climate  in  the  analysis  of  average  annual 
cancellation  rates.  Following  this  suggestion,  Table  7  displays  average  annual 
cancellation  rates  by  acquisition  regime  and  two  budget  climates — Sharply  Decreasing 
Funding,  and  Stable  or  Increasing  Funding. 
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These  comments  are  based  on  an  unpublished  IDA  working  database  drawn  from  various  US 
government  sources.  We  are  indebted  to  Dr.  Daniel  Cuda  for  providing  these  data. 
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Statistical  analysis  of  the  data  in  Table  7  yields  two  conclusions.  First,  the  average 
annual  cancellation  rate  is  significantly  higher  in  periods  when  acquisition  funding  is 
Sharply  Decreasing  than  it  is  in  periods  of  Stable  or  Increasing  funding.  Second,  while 
there  is  no  significant  association  of  funding  climate  and  acquisition  regime  in  the 
Sharply  Decreasing  funding  climate,  the  average  annual  cancellation  rate  under  the  DAB 
post  AR  was  significantly  higher  than  it  was  under  the  other  three  regimes  for  which  we 
have  observations  in  the  Stable  or  Increasing  climate. 


Table  7.  Average  Number  of  Cancellations  per  Fiscal  Year  by  Acquisition  Regime 
and  an  Alternative  Specification  of  Funding  Climate 


Sharply  Decreasing  Funding 

Stable  or  Increasing  Funding 

Acquisition  Regime 

Period 

Cancellation  Rate 

Period 

Cancellation  Rate 

DSARC 

1970-1982 

0.5/yr. 

(6  in  13  yrs.) 

Post-Carlucci  DSARC 

1986-1989 

2.3/yr. 

(9  in  4  yrs.) 

1983-1985 

0.3/yr. 

(1  in  3  yrs.) 

DAB 

1990-1993 

3.3/yr. 

(13  in  4  yrs.) 

AR 

1994-2000 

0.4/yr. 

(3  in  7  yrs.) 

DAB  post  AR 

2009-2012 

2.8/yr. 

(11  in  4  yrs.) 

2001-2008 

1 ,3/yr. 

(10  in  8  yrs.) 

Note:  Number  of  cancellations  in  parentheses. 


We  found  no  strong  reason  in  the  record  of  changes  in  acquisition  policy  or  process 
or  in  the  Program/Budget  process  that  accounts  for  the  significantly  higher  cancellation 
rate  for  the  DAB  post  AR  during  FY  2001-FY  2007.  Inspection  of  the  list  of  cancelled 
programs  indicates  that  four  of  the  ten  cancellations  were  initiated  by  the  Defense 
Acquisition  Executive  (DAE)  (that  is,  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  (USD(AT&L)))  or  a  Service  Acquisition  Executive  (SAE), 
where  in  the  previous  fifteen  years  the  SARs  attributed  only  two  cancellations  (out  of 
twenty-five)  to  the  DAE  (and  none  to  an  SAE).  Moreover,  some  or  all  of  the  four 
cancellations  that  were  attributed  by  the  SARs  to  a  Service  Secretary  may  have  been 
initiated  by  the  SAE.  This  tends  to  confirm  that  the  higher  cancellation  rate  of  FY  2001- 
FY  2007  is  to  a  significant  extent  attributable  to  the  DAB  post  AR. 

In  comparison  to  AR,  the  DAB  post  AR  exercised  more  active  oversight  of  MDAPs. 
It  also  seems  clear  that  changes  initiated  by  DoD  and  statutory  changes  have  increased 
the  authority  of  the  USD(AT&L).  The  mystery  is  why  the  DAE  and  SAEs  would  have 
acted  during  the  boom  years  FY  2003-FY  2007  to  cancel  a  comparatively  large  number 
of  programs.  It  is  relevant  in  this  regard  to  recall  from  the  preceding  section  that  about  30 
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percent  of  the  cohort  of  programs  that  passed  MS  B  during  FY  2003-FY  2007  was 
eventually  cancelled.  Thus,  while  the  DAB  post  AR  of  FY  2003-FY  2007  may  have  been 
exceptionally  active  in  cancelling  programs  that  passed  MS  II/B  in  earlier  years,  it  did  not 
prove  to  be  exceptionally  able  to  limit  the  number  of  new  starts  to  what  future  budgets 
could  accommodate. 

G.  Conclusions 

McN-W  (2014)’s  pivotal  conclusion  about  PAUC  growth  was  simple:  There  was  no 
statistically  significant  association  between  changes  over  time  in  acquisition  policy  and 
process  and  PAUC  growth,  but  there  was  a  strong  association  between  the  budget  climate 
prevailing  when  a  program  passed  MS  II/B  and  PAUC  growth.  The  conclusions  offered 

here  are  broadly  similar  but  more  complicated;  they  include  both  generalities  and 
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exceptions  to  the  generalities,  and  these  may  be  equally  important." 

Average  cohort  cancellation  rates  are  not  strongly  influenced  by  either  acquisition 
regime  or  funding  climate.  The  exceptions  to  this  rule  are  provided  by  the  Post-Carlucci 
DSARC,  which  had  a: 

•  Significantly  higher  cohort  cancellation  rate  than  the  other  acquisition  regimes 
in  the  bust  period  FY  1987-FY  1989;  and  a 

•  Significantly  higher  average  cohort  cancellation  rate  in  the  bust  period  than  in 
the  boom  period. 

This  exception  deserves  further  exploration  as  it  may  provide  an  excellent  illustration  of  a 
mechanism  that  could  help  limit  the  number  of  new  starts  to  what  likely  future  budgets 
will  support. 

In  contrast,  annual  cancellation  rates,  although  not  strongly  influenced  by 
acquisition  regime,  are  strongly  associated  with  funding  climate.  A  possible  exception  to 
the  first  part  of  this  rule  is  provided  by  the  DAB  post  AR,  which  had  a  significantly 
higher  average  annual  cancellation  rate  than  other  acquisition  regimes  during  periods  of 
Stable  or  Increasing  funding.  There  were  no  exceptions  to  the  second  part  of  the  rule.  The 
complications  and  uncertainties  involved  with  the  conclusions  on  the  effects  of  changes 
in  acquisition  regimes  should  not  be  allowed  to  obscure  this  point.  Sharp  cuts  in  DoD 
funding  are  associated  with  high  annual  cancellation  rates:  about  60  percent  of  all 
cancellations  took  place  in  two  periods  of  large  cuts  in  acquisition  funding. 


McN-W  (2014)  presented  evidence  indicating  that  the  funding  climates  that  a  program  passes  through 
after  MS  II/B  seem  to  have  comparatively  little  influence  on  PAUC  growth.  The  analogous  condition 
evidentially  does  not  hold  for  cancellations,  and  it  is  basically  for  that  reason  that  the  analysis  of 
cancellation  is  more  complicated  than  the  analysis  of  PAUC  growth. 
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Although  at  first  glance  it  might  seem  that  these  two  conclusions  together  absolve 
the  MS  II/B  process  from  responsibility  for  cancellations,  they  do  not.  It  is  important  to 
distinguish  three  categories  of  cancellations.  First,  some  programs  are  cancelled  because 
the  threat  or  technology  has  changed.  Cancellations  legitimately  in  this  category  typically 
should  be  viewed  as  an  appropriate  response  to  altered  circumstance.  Second,  some  of  the 
programs  cancelled  at  times  when  funding  for  acquisition  was  sharply  contracting  have 
shown  signs  of  failing — badly  slipped  schedules,  high  cost  growth,  and  continued 
technical  difficulties.  These  failures  probably  can  be  counted  as  failures  of  the  acquisition 
process  even  though  budgetary  considerations  dictated  the  timing  of  the  cancellations. 
Third,  some  programs  cancelled  should  not  have  been  started  when  they  were  because 
they  could  not  be  sustained  at  the  future  funding  levels  that  then  could  be  reasonably 
expected.  This  paper  suggests  that  responsibility  for  this  third  category  of  cancellations 
rests  with  the  means,  or  lack  of  effective  means,  for  relating  DoD’s  acquisition  process 
and  the  resource  allocation  process.29 

The  conclusions  offered  by  this  paper  primarily  are  useful  because  they  direct 
attention  to  changes  in  funding  levels  and  the  resource  allocation  process  whose 
centrality  has  not  been  recognized  in  discussions  of  acquisition  reform.  Ahention 
typically  has  focused  on  acquisition  policy  and  process.  While  changes  in  acquisition 
policy  and  process  are  quite  possibly  justified  on  other  grounds,  the  many  that  have  been 
made  over  the  past  45  years  seem  to  have  influenced  cancellations,  at  most,  episodically. 
In  contrast,  changes  in  funding  climate  are  strongly  associated  with  cancellations,  but  the 
role  of  the  resource  allocation  process  has  received  limited  attention  in  discussions  of 
acquisition  refonn.  It  arguably  is  crucial.  DoD  force  structure,  the  capabilities  that  the 
Department  was  expected  to  provide,  and  funding  were  inconsistent  during  the  1970s  and 
for  more  than  a  decade  after  the  end  of  the  Cold  War.  That  inconsistency,  rather  than 
flaws  in  acquisition  policy  or  process,  is  the  context  in  which  most  cancellations  arise  and 
presumably  is  a  major  factor  to  be  considered  in  designing  proposals  for  improved 
outcomes. 


Examples  of  each  of  these  categories  of  cancellations  appeared  in  the  discussion:  Cancellations  of  some 
systems  following  the  end  of  the  Cold  War,  cancellation  of  seven  of  the  FY  1987-FY  1989  cohort 
during  FY  1990-FY  1993,  the  cancellation  in  FY  2010-FY  2011  of  four  systems  from  the  FY  1998— 
FY  2002  cohort  that  on  average  had  been  in  SDD  nearly  10  years.  It  would  be  useful  to  know  more 
about  the  characteristics  of  programs  that  were  cancelled  and  how  they  compare  to  contemporaneous 
programs  that  were  not  cancelled. 
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